1. Field of Invention
The invention relates to a method for computer-assisted stabilization of images, particularly electronically recorded images (i.e., using a video camera) with respect to rotations of the camera about a random axis and by randomly large angles. The invention also relates to image recording systems using a fisheye lens for (hemi)spherical images, particularly cameras with a 3600 all-around view.
2. Description of the Prior Art
Electronic image stabilization methods are known and can roughly be broken down into two categories, namely (1) compensation devices for the actual camera or for the optical path from object to camera screen and (2) computer-assisted correction of the recorded images, which inter alia presupposes an electronic acquisition of the images and requires reprocessing.
Examples for devices of the first category are described in DE 43 42 717 A1 or DE 196 18 979 A1. These are particularly suitable for compensating blurring in the case of photographs on film material.
However, the second category has also acquired practical significance with modern digital cameras and increased computer capacity. Existing blur compensation methods, such as electronic image stabilization in Handy Cams, can only effect a limited compensation due to the limited angular aperture of perspective cameras, and as a result of perspective transformation the image is distorted. Upon tilting the camera by more than the angular aperture, it is no longer possible to perform any compensation.
It represents a fundamental difficulty of electronic reprocessing that a real image of a (three-dimensional) scene is only acquired by a two-dimensional pixel array. To be able to conclude from a rotation or translation of the camera, which has taken place in the meantime solely from a sequence of images, it is necessary to numerically reconstruct the 3D scene, and even nowadays, this involves considerable effort and expenditure, particularly in real time.
Thus, for the reconstruction of scenes, use of additional external sensors is sometimes made. It is known to equip a camera with a fixed mounted acceleration gyro sensor in order to record the rotation of the camera simultaneously with the images (e.g. Suya You, Ulrich Neumann, and Ronald Azuma: Orientation Tracking for Outdoor Augmented Reality Registration, IEEE Computer Graphics and Applications 19, 6 (November/December 1999), 36-42). Through accelerometers (gyroscope principle) the rotation sensor measures the relative change to the 3D orientation between different recording times. In addition, use is made of further absolute measured quantities, such as the magnetic North Pole and gravitation (plumb line) for producing an absolute orientation reference. Through the integration of the relative change over time, an absolute 3D orientation is determined. The sensor rotation can be described by three angles (Rx, Ry, Rz) about the three space axes (X, Y, Z) (Euler angle, see FIG. 1a) or by rotating about a rotation axis in space (see FIG. 1b, axis-angle representation (A, α), also quaternion representation).
This makes it possible to decouple the camera rotation and translation, i.e., both can be determined separately.
A hemispherical image recording system, particularly a camera with a fisheye lens, is also known and forms a hemisphere of the space on a circular disk in the image plane. 2D image representation takes place in angle coordinates over the unit sphere. The image center images the pole of the hemisphere. The spherical surface is parameterized by two angles running along the degree of longitude (θ) and the degree of latitude (φ) of the sphere (see FIG. 2a). A point (θ, φ) of the spherical surface is imaged by the spherical imaging on the image plane (x, y), in which (x0, y0) images the pole (optical axis) (see FIG. 2b).
For example, U.S. Pat. No. 6,002,430 discloses a spherical image recording system imaging the entire surrounding space, in each case a half-space is recorded by two hemispherical cameras fitted back to back. Due to the non-zero spacing of the two cameras, it is obvious to use fisheye lenses with a 190° viewing angle in order to be able to seamlessly assemble the two images to form a complete all-around image. However, there is always (1) a “dead zone” of the device in the immediate vicinity of the double camera and which cannot be perceived, together with (2) a “near zone” in which the parallax shift of the two cameras is noticeable. The camera center spacing, however, plays no part when the scene points are sufficiently far apart.
In the field of so-called “augmented vision”, the aim is to fade additional information into the visual field of people, as in the case of complex operations. The people are equipped with transparent display lenses or spectacles which are able to fade texts or graphics into a real view comparable with the heads-up display (HUD) known from military aircraft. In order to be able to automatically determine the necessary precise knowledge of the position of the lens in 3D space, it is possible to rigidly connect to the lens in all-around viewing cameras. The real time analysis, i.e., the movement of the known marking points in the recorded image, makes it possible to determine the position of the lens in the operating space. However, for external applications, much more complex image analyses are needed to be able to obviate the use of such markings.
It is helpful to be able to separately analyze the rotation and translation influences on the image of the unknown scene. It is therefore obvious to also use rotation sensors here. However, no method has been published as to how it is possible to use additional rotation data for real time rotation stabilization of the spherical image.
The problem of the invention is, therefore, to real time process the electronically recorded image of a spherical camera in such a way that when displayed on a screen (i.e., of a PC) the camera remains invariant with respect to random rotations.